Description

This PR adds a semantic load-balancing algorithm to ai-proxy-multi, so a route can pick an LLM instance by what the request is asking for rather than by weight (roundrobin) or by a hash key (chash).

Problem

ai-proxy-multi today can spread traffic across several LLM instances, but the choice is blind to the request. In practice the instances behind one route are rarely interchangeable — a frontier model, a small cheap model and a domain-tuned model differ by an order of magnitude in cost and latency. Today the only ways to send the right prompt to the right model are:

• make the client choose, by sending a different model (leaks backend topology into every caller, and every model swap becomes a client-side change), or
• put each model on its own route and select by URI/header (the caller must already know which model it wants — the gateway is not deciding anything), or
• run a classifier service in front of the gateway (an extra hop, an extra deployment, and its own failure mode).

None of these let a single endpoint accept "model": "auto" and route on meaning.

Proposal

Let each instance describe, in natural language, the kind of prompt it should serve. The gateway embeds those descriptions once, embeds the incoming prompt, and picks the closest instance by cosine similarity.

                       ┌──────────────────────────────────┐
 POST /chat            │ ai-proxy-multi (algorithm=semantic)
 {"model":"auto",      │                                  │
  "messages":[…]}  ──► │  1. extract last user message    │
                       │  2. embed it  ───────────────────┼──► embeddings endpoint
                       │  3. cosine vs each instance's    │    (reference vectors
                       │     reference vectors            │     embedded once,
                       │  4. best score ≥ threshold ?     │     cached per conf version)
                       │       yes → that instance        │
                       │       no  → catchall instance    │
                       └──────────────┬───────────────────┘
                                      ▼
                             chosen LLM instance

Key properties:

• One embedding call per request. Instance examples are embedded in a single batch and cached in an lrucache keyed by route + plugin config version, so a config change invalidates immediately and a steady-state request only embeds the prompt.
• No vector database. There are at most a handful of reference vectors per route; cosine similarity is computed in-process.
• Fail-open. Semantic routing is an optimisation, never a new failure mode. Every error path — embedding endpoint unreachable, non-200, malformed or null vector, non-numeric component, missing index, dimension mismatch (e.g. the embedding model was changed underneath), lrucache error — falls back to the catchall instance (or the first instance if none is marked). The routing path cannot return 500 because embeddings are unavailable.

Use cases

1. Cost routing — send hard prompts to the expensive model, everything else to the cheap one.

The single most common motivation. Coding and reasoning go to a frontier model; translation and summarisation go to a small one; anything that matches neither still gets served.

{
  "balancer": { "algorithm": "semantic", "threshold": 0.5 },
  "embeddings": {
    "provider": "openai",
    "model": "text-embedding-3-small",
    "auth": { "header": { "Authorization": "Bearer $EMBEDDING_KEY" } }
  },
  "instances": [
    {
      "name": "reasoning", "provider": "openai", "weight": 1,
      "options": { "model": "gpt-4o" },
      "auth": { "header": { "Authorization": "Bearer $KEY" } },
      "examples": ["write a python function", "debug this stack trace", "explain this algorithm"]
    },
    {
      "name": "cheap", "provider": "openai", "weight": 1,
      "options": { "model": "gpt-4o-mini" },
      "auth": { "header": { "Authorization": "Bearer $KEY" } },
      "examples": ["translate this sentence", "summarize this paragraph"]
    },
    {
      "name": "default", "provider": "openai", "weight": 1,
      "options": { "model": "gpt-4o-mini" },
      "auth": { "header": { "Authorization": "Bearer $KEY" } },
      "catchall": true
    }
  ]
}

Clients always POST "model": "auto". Swapping gpt-4o for a newer model, or re-balancing what counts as "hard", is a gateway config change — no client ships a new build.

2. Domain specialisation — route to fine-tuned or self-hosted models per subject.

A support assistant fronted by one endpoint, backed by a billing-tuned model, a technical-troubleshooting model, and a general model as catchall. The client does not need to know the taxonomy, and adding a fourth domain is one more instance with a few examples.

3. Data-residency / compliance routing.

Prompts that look like they carry regulated content ("examples": ["my social security number is", "here is the patient's diagnosis"]) route to a self-hosted instance with a tight threshold; everything else goes to the public API via catchall. Note this is best-effort routing, not a DLP control — it complements, and does not replace, a content-inspection plugin.

4. Progressive model migration.

Point a new model at a narrow set of examples and give it a high per-instance threshold, so it only receives prompts it is confidently a match for. Widen the examples (or lower the threshold) as confidence grows, and drop the old instance when its share reaches zero. expose_scores shows the score distribution while tuning.

Configuration

The embedding request is issued through the existing ai-provider layer (openai-embeddings capability), so it reuses the same endpoint resolution, authentication and HTTP transport as the chat path rather than a second HTTP client. It is a self-contained sidecar call: the client's own request headers are deliberately not forwarded to the embedding provider.

Debugging the routing decision

balancer.expose_scores (default false) makes the plugin report how each instance scored and which one it picked:

X-AI-Semantic-Route: code
X-AI-Semantic-Scores: code:0.8213,translate:0.1904

X-AI-Semantic-Scores lists every instance that has examples, highest score first — not just the winner, so you can see how close the runners-up came. The catchall has no examples, hence no similarity score, and does not appear.

When nothing clears the threshold, the pick becomes fallback while the scores still show how close each one got — which is exactly what you need to calibrate threshold:

X-AI-Semantic-Route: fallback
X-AI-Semantic-Scores: code:0.3057,translate:0.2884

If the headers are absent while expose_scores is on, the embedding step itself failed and the request was served by the catchall before any score was computed.

Since these headers reveal instance names to the caller, they are meant for tuning, not production. The same score list is always logged at warn on the fallback path, with no configuration and nothing exposed to clients:

semantic routing: no instance cleared threshold (scores: code:0.3057,translate:0.2884), falling back

Alternatives considered

• LLM-based intent classification (ask a small model to label the prompt). Higher accuracy on nuanced intents, but adds a full generation round-trip on every request — typically hundreds of ms and a per-request token cost, versus a single embedding call. Also gives a non-deterministic label, which is awkward to threshold.
• Vector database. Unnecessary at this cardinality: a route has a handful of instances and at most 64 examples each. An in-process cosine over a cached, normalized matrix avoids a new dependency and a network hop.
• Local embedding model in the gateway. Would remove the network call, but pulls a model runtime into the data plane. Deferred; the embeddings block already allows pointing at a self-hosted OpenAI-compatible embedding server.

Known limitation

semantic selects an instance but does not participate in health checks or fallback_strategy / retry: if the chosen instance's upstream then fails, that failure is returned to the client. It only falls back when routing is inconclusive (no instance clears its threshold) or embedding fails. Combining semantic selection with health-aware failover is a follow-up; the behaviour is stated in the schema description and the docs so it is not a surprise.

Tests

t/plugin/ai-proxy-semantic.t (similarity math, incl. zero-vector and empty-aggregate edges), t/plugin/ai-proxy-semantic-schema.t (6 negative schema cases), t/plugin/ai-proxy-semantic-routing.t (14 cases): routing by intent, catchall fallback, and one case per fail-open guard — malformed (null) vector, non-table 200 body, non-200 response, dimension mismatch, reference-embedding failure — plus multimodal content arrays and a regression test asserting the client's Cookie never reaches the embedding endpoint.

Which issue(s) this PR fixes:

N/A — new feature.

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change (docs/en and docs/zh)
• I have verified that this change is backward compatible (existing roundrobin / chash routes are unaffected; every new field is optional and semantic must be opted into)